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to learning science, math, technology, and geography. The children 
participate as scientists investigating environmental problems using NASA 
(National Aeronautics Space Administration) aeronautics and airborne remote 
sensing data. The general PBL process, characteristics, and impacts on 
science education and K-12 students served as the theoretical foundation for 
designing the Web-based KaAMS PBL lesson plans. The conceptual framework of 
KaAMS, including KaAMS the PBL model and general characteristics of KaAMS 
learning process emerged from this literature. Twelve lesson plans were 
developed and tested in the classroom. Formative evaluation results are 
presented. The paper concludes that through their high quality materials, 
NASA can make an impact on science in the classroom, which in combination 
with KaAMS strategies can change teaching practice and impress middle school 
students with the importance of and strategies for conducting "good science. 
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Designing Web-Based Science Lesson Plans that Use Problem-Based 
Learning to Inspire Middle School Kids: KaAMS (Kids as Airborne Mission 

Scientists) 



Tiffany A. Koszalka, PhD. Barbara Grabowski, Ph.D. Younghoon Kim 

Syracuse University Penn State University Penn State University 

Abstract 

Problem-based learning (PBL) has great potential for inspiring K-12 learning. KaAMS 
(Kids as Airborne Mission Scientists), an example of PBL, was designed to help teachers 
inspire middle school students to learn science, math, technology and geography. The 
kids participate as scientists investigating environmental problems using NASA 
aeronautics and airborne remote sensing data. The general PBL process, characteristics, 
and impacts on science education and K-12 students served as the theoretical foundation 
for designing the web-based KaAMS PBL lesson plans. The conceptual framework of 
KaAMS, including KaAMS the problem-based learning model and general 
characteristics of KaAMS learning process emerged from this literature. Twelve lessons 
plans were developed and tested in the classroom. Formative evaluation results are 
presented. 



Introduction 

Kids can be motivated and inspired by making direct contributions to solving real 
scientific problems. Can teachers be inspired too? Through a PBL approach, KaAMS, a 
NASA funded project guides teachers to take middle school children on live and past 
NASA airborne missions to collect data to study two environmental problems. The 
ultimate goal of the project was to inspire kids to learn and develop a career interest in 
science, math, technology and geography by their participating as scientists in activities 
punctuated by “bursts” of interactive events culminating in the analysis of data from 
NASA airborne missions. The goal is realized by providing a variety of learning 
resources to teachers to use with middle school children. 

The KaAMS conceptual model (see figure 1) was designed to represent a holistic 
framework for the project. The relationships among the varied components (e.g., NASA 
resources, Web-Enhanced Learning Strategies Interface, NASA Mission, web-based PBL 
learning processes, characteristics of middle school kids, and National Science, Math, 
Technology and Geography Standards) are shown in this figure. In the center portion, the 
four phases of the KaAMS PBL process include the problem scenario, propose ideas, 
conduct the mission, and propose solutions. These phases were selected based on a 
review of both the problem based learning literature and the scientific process. 

Based on this conceptual model of KaAMS, four PBL modules addressing two 
different environmental problems-active lava flows and the health of the coral reefs in 
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Hawaii were developed. The modules consisted of new lesson plans that could be used 
flexibly by many teachers, and tap into existing NASA and other web resources. The 
goal was to harness those resources that exist rather than create totally new ones. 



•From concrete to 
abstract thinking 
•Curious on a wide 
range of topics, few 
of which are 
sustained 

•Prefer active over 
passive learning 
•Respond positively 
to participating in 
real life learning 
situations 

•Are inquisitive and 
challenge adults 
•Desire recognition 




Figure 1. The KaAMS Model 

Conceptual Framework of KaAMS Model 

Foundation and Components of KaAMS Model 

Conceptually, the KaAMS framework, as shown in Figure 1, is built upon the 
premise and foundation that among all NASA web resources from all aspects of the 
agency, a multitude of resources can be used in the classroom. These resources are 
filtered through a second-level premise, which is the Web-Enhanced Learning Strategies 
(WELES) interface (Koszalka, Grabowski, & McCarthy, 2000). This interface helps to 
sift through the available resources for elements and composite sites that are appropriate 
for use by middle school teachers and students. These resources are then used in four 
parts of a lesson plan — frame/inform/explore/try. The third premise is that teachers can 
use real web resources from real NASA missions in a problem based lesson format. 
Finally, these three levels of resources are harvested as part of the KaAMS PBL lesson 
plans. 



Students are presented with an environmental problem for which NASA had 
collected airborne data on a previous mission. They begin a series of problem-solving 
lessons from which they develop content and applied knowledge by participating in 
problem solving activities. Through a series of framing and informing activities, students 
search for additional information on the problem, develop an understanding of the 
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science of the problem, and propose a solutions for conducting a mission that will 
provide remote sensing data to solve the problem. Students become involved in “bursts” 
of activities to conduct a mission, collect and analyze data. Finally, students summarize 
their findings in several different ways and “go public” to share what they have learned 
with classmate and/or other outside their classroom. One important note is that the 
students participate in reflective activities throughout the entire process. 

Links to Middle School Kids 

Also evident in the design were the following key characteristics we found about 
middle school students (This We Believe, 2000). 

• Moving from concrete to abstract thinking 

• Curious on a wide range of topics, few of which are sustained 

• Prefer active over passive learning 

• Respond positively to participating in real life learning situations 

• Are inquisitive and challenge adults 

• Desire recognition 



Each of the problem-based learning modules involves students in active, concrete 
activity bursts during which the students reflect on their learning. 



Links to National Standards 

To maintain the link to the National Standards, we have completed an analysis of 
the NSTA/NRC standards and the AAAS Project 2061 Benchmarks to target in the 
KaAMS Project. Each lesson plan links to the specific national education standards that 
might be satisfied by completing the lesson activities 

Flexibility of KaAMS Learning process 

Since flexibility is very important to maximize the usability of this site, we have 
designed the site for the teacher. The framework is constructed so that the teacher is in 
control of how much and what types of the available activities that his or her students 
actually see. He or she can start from Phase 1 and proceed to Phase 4, or he or she can 
just go the activities of Phase 3, for example. 

Understandings from Problem-Based Learning 

The conceptual framework of the problem-based learning model for KaAMS is 
based on the perspectives from the problem-based learning literature. Problem-based 
learning as an instructional model is associated with the new learning-centered paradigm 
(e.g., Reigeluth, 1999). PBL, in general, encourages the students to develop deep 
understanding within a knowledge domain and problem solving skills by engaging them 
in the learning process and activities to solve real world, authentic problems (Duffy & 
Cunningham, 1996; Hmelo & Evensen, 2000). According to PBL researchers (Barrows, 
1986, 1992; Hmelo & Evensen, 2000; Savery & Duffy, 1995; Schwartz, et al, 1999), PBL 
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include the following six key characteristics. Real world problem as the learning context: 
real-world problems with a motivational context drive students’ learning. A real world 
problem is used as a focus or stimulus for the student to get involved in learning activity. 
Student generated learning goals: given a problem space, students generate their own 
learning goals by questioning what they know, what they don’t know but need to know, 
and how to know it. Student access to multiple learning resources: multiple learning 
resources include print, electronic and humans. With access to rich and varied 
information, students are able to develop a deep understanding about the content related 
to the problem so that they may apply that knowledge to the problem at hand. Students 
as active problem solvers — experimenting, gathering data, reflecting, collaborating and 
communicating: students as active problem solvers work with their peers, teachers, and 
experts to share their different perspectives. By engaging students, they must exhibit 
problem solving skills, reflective thinking skills, and collaboration and communication 
skills. While being engaged in the process, students can assume the roles of various 
participants involved. Finally, teacher as coach or facilitator: teachers play a role as 
coaches or facilitators that support students’ learning and problem solving activity, rather 
than directly teaching entirely what students should know and how students should solve 
a problem. 

The PBL learning process, cycles students through the following five learning 
stages (Barrows, 1986, 1992; West, 1992): students are presented with a problem, 
students develop a plan— generate what they know and what they need to know and list 
possible actions, students collect information, analyze data and present and share 
solutions. 

• Students are presented with a problem — A real world problem is presented to 
students, and students get clarification about the problem. 

• Students develop a plan: Generate what they know and what they need to know, 
and list possible actions — Students actively define problems and generate what 
they know and what they need to know based on their prior knowledge and 
experience. They are encouraged to identify learning issues or knowledge 
necessary to construct an understanding about how to solve problems. Students 
then discuss and generate strategies and activities for solving the problem. 

• Collect information — the student engages in gathering information from available 
learning resources ranging from print-based materials, electronic and human 
resources (e.g., peer, teacher, and expert). 

• Analyze data— after gathering the information, they analyze and evaluate 
information in terms of what is most useful or what is not useful to solve the 
problem. They discuss and negotiate their perspectives about alternative solutions 
with peers, their teacher, and experts 

• Present and share solutions — finally, students propose their solutions, share them 
with their peers and experts who might provide different perspectives to the 
solutions, and revise their solution based on feedback from their peers or experts. 

PBL has been implemented in diverse content domains such as medical education, 
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business education, social education, and science education. Many researchers have 
investigated the effectiveness of PBL, especially in medical education. According to 
Hmelo and Evensen (2000), the results of PBL research in medical education have 
showed that students who engaged in PBL were able to solve problems and transfer their 
learning better than students who studied under a conventional learning approach. 
However, there have been few PBL research studies focused on middle or high school 
science education. In one study, West (1992) addressed the benefits of PBL used in 
secondary school science classrooms. He concluded that PBL could be an effective 
instructional strategy or technique to stimulate students’ science interest, enhance 
knowledge construction and problem solving skills, and integrate science with other 
knowledge domain. 

Through the PBL literature, as described above, some design considerations and 
strategies were incorporated into the KaAMS learning process. The following section 
illustrates these design considerations and strategies. 

Incorporating PBL and the Scientific Process into KaAMS 

The PBL learning process defined from the literature and the scientific process used 
by NASA scientists for solving environmental problems are similar, making the creation 
of PBL lesson plans for middle school kids using NASA missions a natural development. 
This cross over also made explaining the learning process to content experts very easy. 

First, the PBL process was streamlined into four learning phases in the conceptual 
model that naturally match the evolution of a lesson plan — frame, inform, explore and try 
from the Web-Enhanced Learning Environment strategies framework (Grabowski, 
Koszalka, & McCarthy, 1999). See figure 1. KaAMS phase 1 is to present the scenario. 
At this point in the process, learning is framed by the context of the problem. Students 
obtain clarification about the problem and what their task is. During phase 2, students 
propose ideas and search for information. This is the inform phase. Students create a 
plan of study and review background information that will help them clarify issues 
surrounding the problem so that they can plan for conducting a NASA mission that will 
generate data to help them solve the problem. The third and fourth PBL phases were 
combined into one, conducting the mission. During this learning phase, students explore 
by collecting and analyzing actual NASA data. In the last phase, the students try out their 
knowledge by proposing solutions in a public forum (Go public) (Schwartz, Lin, Brophy 
and Bransford, 1999). 

The scientific process follows the same path, but with a purpose of solving a real 
problem, versus having learning as its primary goal. In this process, scientists identify the 
problem, research ideas and develop a plan for investigation, collect and analyze data and 
report their findings. The phases of the PBL process from the literature, the learning 
phases of KaAMS, and the scientific process are mapped together in Table 1. 

The last column exemplifies how these phases were actualized in the KaAMS 
project. 




KaAMS Problem-Based Learning 6 



Table 1. Mapping KaAMS onto PBL and the Scientific Process 



PBL 

Process 


Scientific 

Process 


KaAMS 


Design Examples 


Problem 

Presentation 

and 

Clarification 


Identify 

Problem 


Problem 

Scenario 


• Two problems — finding lava flows 
for the Pacific Disaster Center, and 
determining if the coral reef need 
protection for a real Congressional 
Executive Order 


Plan 

Development 

-what 
they know 

-what 
they need to 
know 

-list 

actions 


Research 
ideas and 
Develop plan 
for 

investigation 


Propose 

Ideas/Search 

for 

Information 


• Activity Sheets as Who, What, 
When, Where, Why and How 
questions to determine what the 
students know 

• Students complete reflection 
journals 

• Find information using existing 
NASA web resources 

• Participate in activity bursts to 
explore new concepts 


Collect 

Information 


Collect 

Data 


Conduct 
Mission and 
Collect Data 


• Students think critically about 
which aircraft can run their mission 
and select from several possibilities 

• Students plan the actual mission and 
compare it to the actual NASA 
mission 


Analyze 

Information 


Analyze 

Data 


Analyze 

Data 


• Students use actual data from the 
live missions to draw conclusions 

• Students use guidelines from NASA 
scientists for interpretation of their 
data 


Present 

Solutions 


Report 

Findings 


Propose 

Solution: 

(Go 

Public) 


• Write a report to the Pacific 
Disaster Center, 

• Make recommendations to the 
President in response to his 
executive order 
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Example of KaAMS “Active Lava Flows in Hawaii” Lesson Plans 

Lesson Plans: Mission I - Active Lava Flows in Hawaii 

This mission, Active Lava Flow in Hawaii, puts the students in the role of Airborne 
Remote Sensing scientists concerned with identifying where the lava flows are active on 
the Kilauea volcano. This mission consists of 12 lesson plans that are organized in four 
learning stages associated with the processes of PBL and scientific inquiry, as described 
above. 

Figure 2 shows the main screen of this mission that includes a visual representation 
of its learning cycle to help teachers understand the relationships between lesson plans 
and where each fits within the entire learning process of KaAMS. 
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Resources 
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Community 


> 


Links 
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FAQ 


> 





Move your cur a or over the diagram to the loft to 
get a short description, or scroll down the page 
to get a slightly longer description of each losoon 
plan. Click on the circles in tho diagram to the left 
or on the titles below to go to tho looson plan 
overview. This overview contains a thorough 
description of each lesson plan and provides an 
option to print or viow a detailed version of the 
lesson plan. 

You can also print the accompanying student 
guide for either the aeronautics or remote sensing 
pathway by clicking on the student guide images 
to the bottom left 



What is AisfesvEfe JEte 

Summary. This lesson plan provides an overall 
context for KaAMS. First* it provides the 
students with an authentic and motivating 
problem to investigate. Second* it provides an 
explanation of the final project expectations fra 
this serios of lesson plans. Third* it prompts 
students to bo gin the process of exploring the 
overall problem by having them develop an 
understanding of key concepts in airborne 
remote sensing. Finally, it provides students with 
a framework for being scientists Who " do* 
science rather than just learning about science. 



r 



Figure 2. Main Screen of Mission I - Active Lava Flow in Hawaii 



Learning Phase 1: Problem Scenario 

To motivate students and promote their engage in an authentic problem, the 
problem scenario requests that students function as airborne mission scientists to 
investigate the location of active lava flows on the Kilauea volcano, one of the world's 
most active volcanoes (see figure 3). The problem scenario prompts students to begin the 
process of exploring the overall problem by having them develop an understanding of 
key concepts such as aeronautics, remote sensing, and airborne remote sensing. It also 
provides students with a sense of being airborne mission scientists who use aeronautics 
principles and remote sensing data to study an environmental problem of the earth. 




9 



BEST COPY AVAILABLE 





KaAMS Problem-Based Learning 8 



U ~~’ Mission request letter (AkS-1) 



4 'S 



Psxdlc ilsastor .Management; Agency S 



v\ 






NASA Airboms Mis cion Science Division 
NASA Dryden Flight Research Center 
Edwards, Californio 93536 



Deer eiibome mission scientist. 

Our agency studies many types of natural hazards in the Pacific region including tsunamis, landslides, earthquakes, and volcanic 
eruptions. Kilauea, on erupting basaltic shield volcano on the island of Hawaii, has been continuously active since January, 1583. Since the 
beginning of the eruption, over 180 homeo in many communities have been destroyed. In ad d i t io n , the eruption has effected Lho island's road 
network. The main Chain of Craters Road linking the area near the top of Kilauea with the community ofKalepana has been covered by lava 
causing problems for the residents. Our agency constantly monitors Kilauea m a number of ways. We often utifeo airborne image data to map 
the locations of recent love flow deposits and, more importantly, the areas of the volcano where lava flows are currently active. 

We would like to request your help in locating active lava flows on Kilauea. The knowledge of their location will aid the Agency in 
providing information concerning the location and extent of lava flow activity and support the Agency's on-going effort to re-examine our 
emergency evacuation plans in tha event of substantial eruptions. 



Figure 3. An Example of the Problem Scenario 
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Air bo mo Remote Sensing 
Mission: What do we need 
to know? 
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Teacher Activities Student ArtMdts 

TRY using new knowledge about the types of questions needing to be 
answered to select the best aircraft for a mission to solve the KaAMS 

problem. Semple student responses: 



Break students into several groups, assign at least one member from each of 
the activities completed above into each of these groups, and prompt 
students to generate a complete list of questions that need to be addreesed 
in order to choose the best aircraft for this mission over Kilauea. 

Prompt students to think about important questions needing exploration to 
select the bast aircraft for locating the active leva flows on Kilauea. N ote: 
provide the students with a copy of the Activity sheet: Prrcnrrmtfrmrrhrfc 
fARSM-91 to help them prepare their list, 

* Which questions are tha most important to consider when 9 ole cling 
on aircraft for this mission? Why? 

• What other questions do you think ere important? 

Ash each group to present their question lists and the reasons why they 
generated those questions. Students should refer to the Activity sheet: 
Mfereten Atflnlrei nrr^rr? iARSM-flf, 



• Questions on selecting the appropriate aircraft given 
the remote sensingroquiraments. 

• Questions on outside factors such os weather. 

• Questions on flight planning information. (In line with 
what they just studied). 

Student activity: 

• Generate Hsts of questions. Students should record 
their questions on Activity sheet; MHnfrm ^cflrdbrrq 
qr^yrti'im (ARSM-8V. 

o Factors relating to the aircraft and remote 
sensing instruments), 
o Outside factors such as weather, 
o Factors relating to flight planning. 



Ash. each group to asoees and provide feedback about other groups' 



Figure 4. An Example of “Propose Ideas ” Learning Phase 



Learning Phase 2: Propose Ideas and Search Information 

After studying the problem scenario, the learning stage of “Propose Ideas and 
Search Information” encourages teachers and students to propose their initial ideas of 
what they need to know to solve the problem and ways they might solve the problem (see 
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figure 4). They are encouraged to explore existing NASA web resources to learn the basic 
science necessary to solve the problem. A lesson plan within this learning stage, for 
example, presents a variety of learning activities for students to develop an understanding 
of who airborne mission scientists are, how they explore the world, and how these 
scientists work together. 

To support these activities, a guided reflective journal for students is provided. 
Students are encouraged to write their own reflective journals while participating in 
activity bursts to explore new concepts and principles such as aeronautics, remote 
sensing, the roles of airborne mission scientists, and characteristics of volcanoes and lava 
flows. The reflective journal helps students reflect on what they have learned throughout 
the learning processes of KaAMS. 



Learning Phase 3a: Conduct Mission 

After searching information and developing an understanding of the problem, 
students are given an opportunity to select which NASA aircraft they will use to run their 
mission to collect actual data (see figure 5). In this learning stage, students are prompted 
to think about how data can be collected using airborne remote sensing aircraft by taking 
part in a kite aerial photography activity. To support students’ learning in this learning 
stage, for example, several hand-on activities such as flying a kite, developing film, and 
analyzing data images are provided for students. 



- y - a) jjj £1 1 m gRgTgSj; M - !H & 



Collecting Data 
- Klto Aorta! Photography 
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INFORM 
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Teacher Activities Student Activities 

INFORM that they win mod to split into groups to budd a fcdfl aerial photography 

craft and develop the skills necessary to fly an airborne remote s easing mis cion. Student activities: 



Diride students into groups and have them split their team into two roles: "rig" • Divide students into groups. 

builders end kite 'pilots". See Activity sheet; Team Assignment (CD-I) See Activity sheet: Team Assignment (CD-I) 



• Activity - Building the "rig* 



Students read/ view the explanation for 
budding rig in " how fo * wob cite. 



Out do the rigging students through the procoos of building the "rig* 
using a projection panel to demonstrate the step-by-step process as 
described on the website. 



Students brainstorm mission flight plan and 
necessary piloting skills. 



• Activity - Planning for flying a kite aerial mission 



Prompt “pilot* students to develop a flight plan for the mission and a 
list of skills the "pilot" noods to fly a controlled miss ion. 

Student activity: 

Teacher note; Students may revisit the "Developing the mission flight plan lesson" 

to get ideas about how to develop a plan fbr flying a mission * Students view images from remote sensing 

missions and think about how s dentists 
collect such images. 



Figure 5. An Example of “Collect Data ” Learning Phase 



After collecting the data, students participate in numerous activities to learn how to 
analyze and interpret both visible and infrared remote sensing images (see figure 6). They 
analyze and interpret two actual NASA images about Kilauea volcano to locate the active 
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lava flows. To support these learning activities, rich visual images existed in NASA 
websites and guidelines from NASA scientists are provided for students to analyze and 
interpret their data. These supports help students to understand how to interpret the data 
and then find the location of active lava flows. 



€) H ca FT rn m SF ®T1F 






Analyzing Data 



E*ptoro 

— o — 



EXPLORE 





Teacher Activities Stride at Activities 

EXPLORE concepts related to analyzing remote censing dale by participating in actrvttioo that raako 
usa of visible and non- visible images. 



Bxealc students into 4 small teems end have them explore one of the following activities and record 
thou findings on Activity c he ett Writchlu”; TOT otirpTxnct from (AJB2), You can download 

the entire Ffrnet Kit or only the sections you choose. 



• ti\ : Wbxcfui r/ludvTU) : Matching images with a description of features 

■ P 'X : Fin-1 it (3 7t : Finding and determining the location of features on an image 

• 3a.: Mcncuifi this 0.3. a) : M a osuting distanc a 

» P 3h: CVUtvAurv tfaig. P 4 -b): Measuring features recognition, determining direction 

• Thein ere many forest fires in every year. Satellite image can be used to 
mop the typo of vegetation, sources of water and ora an that are difficult to travel over. In this 
satellite image, various colors show various areas Student can identify the color of forest 
fire by analyzing data image. 

■ PpUosAl activity: You thrum it Students can solve mullipla choice question using 

image reading shills and interpret images using contextual logic. (This activity should be 
considered optional) 



Figure 6. An Example of “Analyze Data ” Learning Phase 



Student activity; 

• Students work in groups to 
complete one of the 
octivitteo. 

m Complete student 



Learning Phase 4: Propose Solution 

After analyzing and interpreting the data, students write results of their 
investigation for the KaAMS mission, locating the active lava flows on Kilauea (see 
figure 7). Each student group presents the best solution and shares it with the other 
students and teachers. After all the solutions are presented, students have an opportunity 
to revise their solutions based on feedback from their peers, teachers, and experts before 
making their final statement. With this statement, they complete their investigation and 
mission. 
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Go Public? 

Presenting Your Results 





FRAME 



Related ©object area; science 

Overall problem Where trs the active lave flows loe »ted on the Kilauee volcano? 

Relationship ofpnofetera In tWa lesson to overall problem. Students have been working through a process to identify is cuts, research 
mf arm Brian, explore new ecientiflc concepts, gather and analyze date, and eutremeriie findings. During Utus lesson students complete the 
research cycle by going pubhc with their findings. This session provides students with an opportunity to summarize the work they have done 
in KaAMS and charo it with others. At this point they ten describe how they researched the issues, what they teamed, end their answer to 
the overall problem 

Es timate d tfaaa required: 1 to 2 class periods (without class lima f at preparing final project) 

Student oatcomas/objectlwo; 

■ A final product that exemplifies the scientific process, content covered in this unit, and conclusions to the problems en countered 

Hole: Up to this pomi, students have been working in groups to document their investigations during throughout the unit. It is your option to 
allow class rime for students to create a poster, web ado. muhimedia project, or scientific report that presents their findings end the work they 
did to reach those conclusions or ensign final project preparation as homewoxk. 

Prerequisite ekllb or knowledge: 

■ Analysie of the remote sensing data 

• Simple internal skillr 



Figure 7. An Example of “Propose Solution ” Learning Phase 



Across each learning Phase of KaAMS, the following characteristics of PBL can be 
found: 



• Authentic, ill-structured problem situation 

• Assumption of roles by the students 

• Reflections about what they know, what they need to know 

• Planning the investigation procedure 

• Access to rich NASA web resources 

• Active investigation 

• Learning activities situated within real NASA missions 

• Reflective thinking exercises 

• Peer and expert collaboration 

• Learner activities/tools in interpreting data gathered 

• NASA scientist support 

• Shared solutions with peers and experts 



Formative Evaluation of KaAMS 

Overall Assessment and Research Strategy of KaAMS 

The assessment strategy for the entire KaAMS project was divided into three major 
phases designed to capture data that would support initial product development (alpha 
testing), on-going resources development and implementation planning (beta testing), and 
the KaAMS impact on the stakeholders in the learning environment (research-impact 
analysis). See Figure 8. The diagram below illustrates the flow of alpha, beta, and 
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research processes used for each of the two major products developed; (1) lava flow and 
(2) coral reef missions. This report summarizes overall data collection methods and 
procedures as well as the findings from the alpha development for the lava flow Mission. 



ALPHA TEST ^ 
RESEARCH TESTING 

- TAPA* * from PA (Alpha) 

- students from PA 

- parents from PA 



BETA TEST (new classrooms) 



RESEARCH 



KaAMS 
formative / 
summative / 
test research 



Baseline 

School 

Environment 



*TAP-Teacher Advisory Panel (A=alpha, B=Beta) 



Baseline 
classroom and 
TAPB teacher 
characteristics 



Baseline 
student and 
parent support 
characteristics 






IMPLEMENT 
(new classrooms) 



RESEARCH 






- TAPB* (Beta) 

- new teachers from PA/CA/HI 

- new students from PA/CA/HI 

Short-term 
follow-up 



- new teachers PA/CA/HI 

- other teachers 

- administrators/school 




- new students PA/ CA/HI 

- new parents from PA/CA/HI 



Figure 8. Overall Assessment and Research Strategy 



Method of Formative Evaluation 

During the Alpha testing, formative and summative evaluation data were collected. 
With project enhancement in mind, data collected from key stakeholders included five 
levels of assessment: (1) reaction, (2) learning gains, (3) performance, (4) education 
system changes, and (5) impact on the greater society. Research protocols were also 
tested to assess their effectiveness in measuring the effects of KaAMS materials on 
teachers, students, and stakeholders in the surrounding community, namely parents. See 
Table 2. 



Table 2: Research questions and assessment 



Research Questions 



Assessment 



• How are teachers using KaAMS and NASA resources? Performance 



• How are teachers changing their teaching practices (e.g. 
teaching strategies, incorporation of NASA resources, etc.) 
over time as a result of using KaAMS and NASA resources? 

• How are student levels of interest in pursuing science-related 
career changing over time as a result of using KaAMS and 
NASA resources? 

• How does the use of KaAMS diffuse to the surrounding school 
system? 



Performance/ 
System Change 

Learning Gains 



Impact on 
Greater Society 
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All formative evaluation instruments were administered throughout the alpha 
testing phases to gather feedback from teachers while preparing for and using the 
KaAMS materials. Interviews, observations, and focus groups were also conducted at 
least once per week with teachers and students during the 6-month alpha classroom trials. 
The summative instruments were administered to teachers at the end of the KaAMS 
classroom trials and final interviews and focus groups were conducted with teachers and 
students. The research instruments were administered to teachers and students for pre- 
and post-test data collection and at and additional 1 -month follow-up period for students. 
Parents were surveyed at the beginning and end of the school year. 

Participants 

Three middle schools in a rural Pennsylvania school district participated in the 
KaAMS alpha test classroom trials, East, West, and Distant. Six different classrooms 
from these schools were actively involved. Four were 6 grade, one was 7 grade, and 
one was an 8 th grade honors class. The six teachers who participated provided data about 
themselves, their classrooms, and success of using KaAMS materials during the alpha 
testing cycle. Teaching experience ranged from 3 to 23 years; initial preferences for 
primary teaching strategies included hands-on activities, collaborative activities, role 
play, and problem-based learning; half of the teachers had moderate success using web 
resources in their classrooms the other half had not used such resources in their 
classrooms. 

Data were collected from a total of 144 students, 82 were boys, 59 girls and 3 did 
not respond to the gender question. On average, the students had a moderate level of 
interest in pursuing science. 

One hundred and fifty three parents of KaAMS students returned surveys indicating 
their initial perceptions of science in their school and child’s success and interest in 
science as well as reporting their highest attained level of education. A majority of the 
parents did not have college degrees, worked in non-science related jobs, and had a 
neutral opinion of their child’s school’s science program. 

Measures and Instruments 

Formative and Summative evaluation: A series of instruments, observation 
protocols, and interview protocols were developed to collect formative and summative 
data from the teachers and students during the alpha testing development cycle. 

Teachers were asked to review the KaAMS lesson plans, prepare to use the lesson 
plans in their classrooms, and complete evaluation surveys after each lesson and at the 
end of the trial indicating ease of use; value of resources, instructions, and assessment 
guidelines provided; success of activities; amount of preparation time; descriptions of the 
classroom activity during KaAMS lessons; and general feelings about using KaAMS for 
teaching and learning. Teachers were also asked to share feedback during interviews and 
focus groups including responses to questions such as: What did you like/not like about 
the supporting website? What parts of the lesson plans did you use - why? What 
additional support materials did you need to use these materials? What additional 
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materials did the students need? What would you change? 

Periodically students were asked, during interviews and focus groups, to respond to 
questions such as: What did you like/not like about the KaAMS activities? How useful 
were the internet sites? What was happening in the classroom during KaAMS? What did 
you learn? and what would have made these activities more useful to you? Observational 
data were collected several times during the classroom trial that lasted between 3 and 6 
months, depending on the classroom teacher. Observation data were collected on how the 
teachers used the materials, how the students participated in the activities, and artifacts 
developed by the teacher or students during the KaAMS lessons. 

Research: The research questions were focused on the teachers, students, and 
parents. Teachers completed an on-line instrument eliciting background information, 
preferences for classroom activities, and attitudes toward the use of web resources in the 
classroom. The instrument was a combination of an attitude survey previously developed 
and validated for similar research (Koszalka, 2000), a series of questions related to 
perceptions of their school’s ability to support the use of internet technology in the 
classroom (McCarthy, Grabowski, & Koszalka, 1998), and preferences for teaching styles 
(Grabowski, Koszalka, & McCarthy, 2000; Koszalka, Grabowski, & McCarthy, 2000). 
This instrument was administered at the beginning of the classroom trial period and the 
end (pre-post test). 

Data were collected on student level of career interest in science, pre-, post, and 1- 
month after using KaAMS materials. Student career interest surveys were purchased from 
the APA. The survey also included a series of questions developed to assess reflective 
thinking (Koszalka, Song, 2001) and gather demographic data. 

Parents were asked to complete surveys at the beginning and end of the school year 
to assess their perceptions of their child’s school’s science program. The questions were 
taken from previous research on measuring parents’ perceptions of school programs. 

Results 

The initial formative feedback provided guidance in designing support structures 
for the KaAMS website that helped the alpha teachers connect NASA science to their 
curriculum and prompt active student involvement, as scientists, during science class. 
The results from the formative and summative evaluation resulted in: development of 
enhanced lesson plan structures for the KaAMS website, new content support for teachers 
that strengthened the relationship between the overall problem scenario and learning 
activities, further instructions to ‘coach’ teachers in using PBL, web technology, and 
activities that prompt student reflection, stronger ties between lesson plans and national 
education standards and curriculum requirements, and enhanced activities that will better 
meet kids’ needs. 

The initial research findings from the pilot classrooms were very encouraging. 
Although caution is warranted in interpreting these results, analysis of the research data 
collected during the alpha testing cycle showed significant, yet minor changes in 
teachers, students, and parents after the use of KaAMS in the classroom. Table 3 
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summarizes research findings in accordance with the KaAMS project research questions: 
Table 3. Research questions and Alpha Preliminary Findings 



Research Ouestions 


Alpha Preliminary Findings 


• How are teachers using KaAMS 
and NASA resources? 


• Teachers noted the flexibility of KaAMS 

resources and used them in a variety of ways to 
enhance or change the way they teach. 


• How are teachers changing their 
teaching practices over time as a 
result of using KaAMS and NASA 
resources? 


• Several of the teachers tried new ways of 
integrating the web and collaborative activities 
into their teaching; changed their preferred 
method of teaching and the types of resources 
they used regularly in their classrooms, and 
their attitudes toward using web resources in 
the classroom. 


• How are student levels of interest 
in pursuing science-related career 
changing over time as a result of 
using KaAMS? 


• Significant increases in student level of science 
career interest 


• How does the use of the KaAMS 
products diffuse to the surrounding 
school system? 


• Parent perceptions of their child’s school’s 
emphasis on science, school’s ability to 
provide good science experiences, and use of 
appropriate science resources were higher at 
the end of the school year than in the 
beginning. 



'We, believe that we are providing teachers with a venue and structure for using 
NASA web-based materials in their classroom in meaningful and contextualized ways 
that will support student knowledge development in the content and processes of science. 
Through their high quality materials, NASA can make an impact on science in the 
classroom, which in combination with KaAMS strategies can change teaching practice, 
impress middle school kids with the importance of and strategies for conducting good 
science — the ultimate goal being to influence career aspirations of these kids toward 
science. 
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